This notebook is a template with each step that you need to complete for the project.
Please fill in your code where there are explicit ? markers in the notebook. You are welcome to add more cells and code as you see fit.
Once you have completed all the code implementations, please export your notebook as a HTML file so the reviews can view your code. Make sure you have all outputs correctly outputted.
File-> Export Notebook As... -> Export Notebook as HTML
There is a writeup to complete as well after all code implememtation is done. Please answer all questions and attach the necessary tables and charts. You can complete the writeup in either markdown or PDF.
Completing the code template and writeup template will cover all of the rubric points for this project.
The rubric contains "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. The stand out suggestions are optional. If you decide to pursue the "stand out suggestions", you can include the code in this notebook and also discuss the results in the writeup file.
Below is example of steps to get the API username and key. Each student will have their own username and key.
kaggle.json and use the username and key.
ml.t3.medium instance (2 vCPU + 4 GiB)Python 3 (MXNet 1.8 Python 3.7 CPU Optimized)!pip install -U pip
!pip install -U setuptools wheel
!pip install -U "mxnet<2.0.0" bokeh==2.0.1
!pip install autogluon --no-cache-dir
# Without --no-cache-dir, smaller aws instances may have trouble installing
# create the .kaggle directory and an empty kaggle.json file
!mkdir -p /root/.kaggle
!touch /root/.kaggle/kaggle.json
!chmod 600 /root/.kaggle/kaggle.json
# Fill in your user name and key from creating the kaggle account and API token file
import json
kaggle_username = "name"
kaggle_key = "key"
# Save API token the kaggle.json file
with open("/root/.kaggle/kaggle.json", "w") as f:
f.write(json.dumps({"username": kaggle_username, "key": kaggle_key}))
# Download the dataset, it will be in a .zip file so you'll need to unzip it as well.
!kaggle competitions download -c bike-sharing-demand
# If you already downloaded it you can use the -o command to overwrite the file
!unzip -o bike-sharing-demand.zip
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
from autogluon.tabular import TabularPredictor
# Create the train dataset in pandas by reading the csv
# Set the parsing of the datetime column so you can use some of the `dt` features in pandas later
train = pd.read_csv("data/train.csv", parse_dates=["datetime"])
train.head()
| datetime | season | holiday | workingday | weather | temp | atemp | humidity | windspeed | casual | registered | count | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2011-01-01 00:00:00 | 1 | 0 | 0 | 1 | 9.84 | 14.395 | 81 | 0.0 | 3 | 13 | 16 |
| 1 | 2011-01-01 01:00:00 | 1 | 0 | 0 | 1 | 9.02 | 13.635 | 80 | 0.0 | 8 | 32 | 40 |
| 2 | 2011-01-01 02:00:00 | 1 | 0 | 0 | 1 | 9.02 | 13.635 | 80 | 0.0 | 5 | 27 | 32 |
| 3 | 2011-01-01 03:00:00 | 1 | 0 | 0 | 1 | 9.84 | 14.395 | 75 | 0.0 | 3 | 10 | 13 |
| 4 | 2011-01-01 04:00:00 | 1 | 0 | 0 | 1 | 9.84 | 14.395 | 75 | 0.0 | 0 | 1 | 1 |
# Simple output of the train dataset to view some of the min/max/varition of the dataset features.
train.describe()
| season | holiday | workingday | weather | temp | atemp | humidity | windspeed | casual | registered | count | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| count | 10886.000000 | 10886.000000 | 10886.000000 | 10886.000000 | 10886.00000 | 10886.000000 | 10886.000000 | 10886.000000 | 10886.000000 | 10886.000000 | 10886.000000 |
| mean | 2.506614 | 0.028569 | 0.680875 | 1.418427 | 20.23086 | 23.655084 | 61.886460 | 12.799395 | 36.021955 | 155.552177 | 191.574132 |
| std | 1.116174 | 0.166599 | 0.466159 | 0.633839 | 7.79159 | 8.474601 | 19.245033 | 8.164537 | 49.960477 | 151.039033 | 181.144454 |
| min | 1.000000 | 0.000000 | 0.000000 | 1.000000 | 0.82000 | 0.760000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.000000 |
| 25% | 2.000000 | 0.000000 | 0.000000 | 1.000000 | 13.94000 | 16.665000 | 47.000000 | 7.001500 | 4.000000 | 36.000000 | 42.000000 |
| 50% | 3.000000 | 0.000000 | 1.000000 | 1.000000 | 20.50000 | 24.240000 | 62.000000 | 12.998000 | 17.000000 | 118.000000 | 145.000000 |
| 75% | 4.000000 | 0.000000 | 1.000000 | 2.000000 | 26.24000 | 31.060000 | 77.000000 | 16.997900 | 49.000000 | 222.000000 | 284.000000 |
| max | 4.000000 | 1.000000 | 1.000000 | 4.000000 | 41.00000 | 45.455000 | 100.000000 | 56.996900 | 367.000000 | 886.000000 | 977.000000 |
# Create the test pandas dataframe in pandas by reading the csv, remember to parse the datetime!
test = pd.read_csv("data/test.csv", parse_dates=["datetime"])
test.head()
| datetime | season | holiday | workingday | weather | temp | atemp | humidity | windspeed | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 2011-01-20 00:00:00 | 1 | 0 | 1 | 1 | 10.66 | 11.365 | 56 | 26.0027 |
| 1 | 2011-01-20 01:00:00 | 1 | 0 | 1 | 1 | 10.66 | 13.635 | 56 | 0.0000 |
| 2 | 2011-01-20 02:00:00 | 1 | 0 | 1 | 1 | 10.66 | 13.635 | 56 | 0.0000 |
| 3 | 2011-01-20 03:00:00 | 1 | 0 | 1 | 1 | 10.66 | 12.880 | 56 | 11.0014 |
| 4 | 2011-01-20 04:00:00 | 1 | 0 | 1 | 1 | 10.66 | 12.880 | 56 | 11.0014 |
# Same thing as train and test dataset
submission = pd.read_csv("data/sampleSubmission.csv", parse_dates=["datetime"])
submission.head()
| datetime | count | |
|---|---|---|
| 0 | 2011-01-20 00:00:00 | 0 |
| 1 | 2011-01-20 01:00:00 | 0 |
| 2 | 2011-01-20 02:00:00 | 0 |
| 3 | 2011-01-20 03:00:00 | 0 |
| 4 | 2011-01-20 04:00:00 | 0 |
Requirements:
count, so it is the label we are setting.casual and registered columns as they are also not present in the test dataset. root_mean_squared_error as the metric to use for evaluation.best_quality to focus on creating the best model.predictor = TabularPredictor(
label="count",
problem_type="regression",
eval_metric="root_mean_squared_error",
learner_kwargs={"ignored_columns": ["casual", "registered"]}
).fit(
train_data=train,
time_limit=600,
presets="best_quality"
)
No path specified. Models will be saved in: "AutogluonModels/ag-20220403_215705/"
Presets specified: ['best_quality']
Beginning AutoGluon training ... Time limit = 600s
AutoGluon will save models to "AutogluonModels/ag-20220403_215705/"
AutoGluon Version: 0.4.0
Python Version: 3.9.7
Operating System: Darwin
Train Data Rows: 10886
Train Data Columns: 11
Label Column: count
Preprocessing data ...
Using Feature Generators to preprocess the data ...
Dropping user-specified ignored columns: ['casual', 'registered']
Fitting AutoMLPipelineFeatureGenerator...
Available Memory: 391.59 MB
Train Data (Original) Memory Usage: 0.78 MB (0.2% of available memory)
Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
Stage 1 Generators:
Fitting AsTypeFeatureGenerator...
Note: Converting 2 features to boolean dtype as they only contain 2 unique values.
Stage 2 Generators:
Fitting FillNaFeatureGenerator...
Stage 3 Generators:
Fitting IdentityFeatureGenerator...
Fitting DatetimeFeatureGenerator...
Stage 4 Generators:
Fitting DropUniqueFeatureGenerator...
Types of features in original data (raw dtype, special dtypes):
('datetime', []) : 1 | ['datetime']
('float', []) : 3 | ['temp', 'atemp', 'windspeed']
('int', []) : 5 | ['season', 'holiday', 'workingday', 'weather', 'humidity']
Types of features in processed data (raw dtype, special dtypes):
('float', []) : 3 | ['temp', 'atemp', 'windspeed']
('int', []) : 3 | ['season', 'weather', 'humidity']
('int', ['bool']) : 2 | ['holiday', 'workingday']
('int', ['datetime_as_int']) : 5 | ['datetime', 'datetime.year', 'datetime.month', 'datetime.day', 'datetime.dayofweek']
0.1s = Fit runtime
9 features in original data used to generate 13 features in processed data.
Train Data (Processed) Memory Usage: 0.98 MB (0.2% of available memory)
Data preprocessing and feature engineering runtime = 0.09s ...
AutoGluon will gauge predictive performance using evaluation metric: 'root_mean_squared_error'
To change this, specify the eval_metric parameter of Predictor()
AutoGluon will fit 2 stack levels (L1 to L2) ...
Fitting 11 L1 models ...
Fitting model: KNeighborsUnif_BAG_L1 ... Training model for up to 399.84s of the 599.91s of remaining time.
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
-101.507 = Validation score (root_mean_squared_error)
0.04s = Training runtime
0.03s = Validation runtime
Fitting model: KNeighborsDist_BAG_L1 ... Training model for up to 399.76s of the 599.83s of remaining time.
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
-84.1058 = Validation score (root_mean_squared_error)
0.02s = Training runtime
0.03s = Validation runtime
Fitting model: LightGBMXT_BAG_L1 ... Training model for up to 399.7s of the 599.77s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBMXT_BAG_L1 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Fitting model: LightGBM_BAG_L1 ... Training model for up to 399.6s of the 599.67s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBM_BAG_L1 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Fitting model: RandomForestMSE_BAG_L1 ... Training model for up to 399.59s of the 599.66s of remaining time.
Warning: Reducing model 'n_estimators' from 300 -> 67 due to low memory. Expected memory usage reduced from 66.52% -> 15.0% of available memory...
-119.0818 = Validation score (root_mean_squared_error)
0.43s = Training runtime
0.06s = Validation runtime
Fitting model: CatBoost_BAG_L1 ... Training model for up to 399.04s of the 599.1s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
-130.4612 = Validation score (root_mean_squared_error)
192.76s = Training runtime
0.03s = Validation runtime
Fitting model: ExtraTreesMSE_BAG_L1 ... Training model for up to 206.17s of the 406.24s of remaining time.
Warning: Reducing model 'n_estimators' from 300 -> 70 due to low memory. Expected memory usage reduced from 63.55% -> 15.0% of available memory...
-126.8015 = Validation score (root_mean_squared_error)
0.26s = Training runtime
0.07s = Validation runtime
Fitting model: NeuralNetFastAI_BAG_L1 ... Training model for up to 205.78s of the 405.85s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
-135.6753 = Validation score (root_mean_squared_error)
56.1s = Training runtime
0.11s = Validation runtime
Fitting model: XGBoost_BAG_L1 ... Training model for up to 149.5s of the 349.57s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 389 MB
Estimated XGB model size: 1 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 391 MB
Estimated XGB model size: 3 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 391 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 390 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 390 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 390 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 390 MB
Estimated XGB model size: 4 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 390 MB
Estimated XGB model size: 3 MB
-245.1743 = Validation score (root_mean_squared_error)
0.17s = Training runtime
0.03s = Validation runtime
Fitting model: NeuralNetTorch_BAG_L1 ... Training model for up to 149.28s of the 349.35s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Ran out of time, stopping training early. (Stopping on epoch 38)
Ran out of time, stopping training early. (Stopping on epoch 41)
Ran out of time, stopping training early. (Stopping on epoch 43)
Ran out of time, stopping training early. (Stopping on epoch 44)
Ran out of time, stopping training early. (Stopping on epoch 47)
Ran out of time, stopping training early. (Stopping on epoch 50)
Ran out of time, stopping training early. (Stopping on epoch 55)
Ran out of time, stopping training early. (Stopping on epoch 67)
-139.9052 = Validation score (root_mean_squared_error)
142.73s = Training runtime
0.14s = Validation runtime
Fitting model: LightGBMLarge_BAG_L1 ... Training model for up to 6.39s of the 206.46s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBMLarge_BAG_L1 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Completed 1/20 k-fold bagging repeats ...
Fitting model: WeightedEnsemble_L2 ... Training model for up to 360.0s of the 206.41s of remaining time.
-84.1058 = Validation score (root_mean_squared_error)
0.15s = Training runtime
0.0s = Validation runtime
Fitting 9 L2 models ...
Fitting model: LightGBMXT_BAG_L2 ... Training model for up to 206.26s of the 206.25s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBMXT_BAG_L2 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Fitting model: LightGBM_BAG_L2 ... Training model for up to 206.23s of the 206.23s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBM_BAG_L2 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Fitting model: RandomForestMSE_BAG_L2 ... Training model for up to 206.21s of the 206.21s of remaining time.
Warning: Reducing model 'n_estimators' from 300 -> 75 due to low memory. Expected memory usage reduced from 59.44% -> 15.0% of available memory...
-54.1643 = Validation score (root_mean_squared_error)
1.24s = Training runtime
0.08s = Validation runtime
Fitting model: CatBoost_BAG_L2 ... Training model for up to 204.82s of the 204.82s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
-55.3827 = Validation score (root_mean_squared_error)
31.92s = Training runtime
0.02s = Validation runtime
Fitting model: ExtraTreesMSE_BAG_L2 ... Training model for up to 172.85s of the 172.84s of remaining time.
Warning: Reducing model 'n_estimators' from 300 -> 73 due to low memory. Expected memory usage reduced from 61.44% -> 15.0% of available memory...
-54.3022 = Validation score (root_mean_squared_error)
0.37s = Training runtime
0.08s = Validation runtime
Fitting model: NeuralNetFastAI_BAG_L2 ... Training model for up to 172.35s of the 172.35s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
-52.281 = Validation score (root_mean_squared_error)
57.07s = Training runtime
0.12s = Validation runtime
Fitting model: XGBoost_BAG_L2 ... Training model for up to 115.08s of the 115.07s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 398 MB
Estimated XGB model size: 1 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 396 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 396 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 398 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 398 MB
Estimated XGB model size: 1 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 397 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 397 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 397 MB
Estimated XGB model size: 8 MB
-239.1839 = Validation score (root_mean_squared_error)
0.26s = Training runtime
0.04s = Validation runtime
Fitting model: NeuralNetTorch_BAG_L2 ... Training model for up to 114.76s of the 114.75s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Ran out of time, stopping training early. (Stopping on epoch 30)
Ran out of time, stopping training early. (Stopping on epoch 31)
Ran out of time, stopping training early. (Stopping on epoch 32)
Ran out of time, stopping training early. (Stopping on epoch 34)
Ran out of time, stopping training early. (Stopping on epoch 35)
Ran out of time, stopping training early. (Stopping on epoch 38)
Ran out of time, stopping training early. (Stopping on epoch 42)
Ran out of time, stopping training early. (Stopping on epoch 51)
-58.4412 = Validation score (root_mean_squared_error)
109.82s = Training runtime
0.15s = Validation runtime
Fitting model: LightGBMLarge_BAG_L2 ... Training model for up to 4.76s of the 4.76s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBMLarge_BAG_L2 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Completed 1/20 k-fold bagging repeats ...
Fitting model: WeightedEnsemble_L3 ... Training model for up to 360.0s of the 4.72s of remaining time.
-51.0252 = Validation score (root_mean_squared_error)
0.12s = Training runtime
0.0s = Validation runtime
AutoGluon training complete, total runtime = 595.41s ... Best model: "WeightedEnsemble_L3"
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("AutogluonModels/ag-20220403_215705/")
predictor.fit_summary()
*** Summary of fit() ***
Estimated performance of each model:
model score_val pred_time_val fit_time pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order
0 WeightedEnsemble_L3 -51.025159 0.771097 451.308120 0.000255 0.119811 3 True 16
1 NeuralNetFastAI_BAG_L2 -52.280977 0.614861 449.583902 0.122451 57.067749 2 True 13
2 RandomForestMSE_BAG_L2 -54.164250 0.570976 393.755368 0.078566 1.239215 2 True 10
3 ExtraTreesMSE_BAG_L2 -54.302187 0.569826 392.881345 0.077416 0.365192 2 True 12
4 CatBoost_BAG_L2 -55.382727 0.515676 424.435747 0.023267 31.919594 2 True 11
5 NeuralNetTorch_BAG_L2 -58.441240 0.644520 502.331243 0.152110 109.815089 2 True 15
6 KNeighborsDist_BAG_L1 -84.105808 0.030788 0.016252 0.030788 0.016252 1 True 2
7 WeightedEnsemble_L2 -84.105808 0.031049 0.164693 0.000261 0.148441 2 True 9
8 KNeighborsUnif_BAG_L1 -101.506986 0.027615 0.040074 0.027615 0.040074 1 True 1
9 RandomForestMSE_BAG_L1 -119.081773 0.062411 0.431252 0.062411 0.431252 1 True 3
10 ExtraTreesMSE_BAG_L1 -126.801500 0.065241 0.261330 0.065241 0.261330 1 True 5
11 CatBoost_BAG_L1 -130.461205 0.031717 192.763849 0.031717 192.763849 1 True 4
12 NeuralNetFastAI_BAG_L1 -135.675261 0.108620 56.096116 0.108620 56.096116 1 True 6
13 NeuralNetTorch_BAG_L1 -139.905160 0.135293 142.733524 0.135293 142.733524 1 True 8
14 XGBoost_BAG_L2 -239.183912 0.532403 392.773360 0.039994 0.257206 2 True 14
15 XGBoost_BAG_L1 -245.174314 0.030725 0.173756 0.030725 0.173756 1 True 7
Number of models trained: 16
Types of models trained:
{'StackerEnsembleModel_CatBoost', 'StackerEnsembleModel_KNN', 'WeightedEnsembleModel', 'StackerEnsembleModel_TabularNeuralNetTorch', 'StackerEnsembleModel_XT', 'StackerEnsembleModel_RF', 'StackerEnsembleModel_XGBoost', 'StackerEnsembleModel_NNFastAiTabular'}
Bagging used: True (with 8 folds)
Multi-layer stack-ensembling used: True (with 3 levels)
Feature Metadata (Processed):
(raw dtype, special dtypes):
('float', []) : 3 | ['temp', 'atemp', 'windspeed']
('int', []) : 3 | ['season', 'weather', 'humidity']
('int', ['bool']) : 2 | ['holiday', 'workingday']
('int', ['datetime_as_int']) : 5 | ['datetime', 'datetime.year', 'datetime.month', 'datetime.day', 'datetime.dayofweek']
Plot summary of models saved to file: AutogluonModels/ag-20220403_215705/SummaryOfModels.html
*** End of fit() summary ***
{'model_types': {'KNeighborsUnif_BAG_L1': 'StackerEnsembleModel_KNN',
'KNeighborsDist_BAG_L1': 'StackerEnsembleModel_KNN',
'RandomForestMSE_BAG_L1': 'StackerEnsembleModel_RF',
'CatBoost_BAG_L1': 'StackerEnsembleModel_CatBoost',
'ExtraTreesMSE_BAG_L1': 'StackerEnsembleModel_XT',
'NeuralNetFastAI_BAG_L1': 'StackerEnsembleModel_NNFastAiTabular',
'XGBoost_BAG_L1': 'StackerEnsembleModel_XGBoost',
'NeuralNetTorch_BAG_L1': 'StackerEnsembleModel_TabularNeuralNetTorch',
'WeightedEnsemble_L2': 'WeightedEnsembleModel',
'RandomForestMSE_BAG_L2': 'StackerEnsembleModel_RF',
'CatBoost_BAG_L2': 'StackerEnsembleModel_CatBoost',
'ExtraTreesMSE_BAG_L2': 'StackerEnsembleModel_XT',
'NeuralNetFastAI_BAG_L2': 'StackerEnsembleModel_NNFastAiTabular',
'XGBoost_BAG_L2': 'StackerEnsembleModel_XGBoost',
'NeuralNetTorch_BAG_L2': 'StackerEnsembleModel_TabularNeuralNetTorch',
'WeightedEnsemble_L3': 'WeightedEnsembleModel'},
'model_performance': {'KNeighborsUnif_BAG_L1': -101.5069858001322,
'KNeighborsDist_BAG_L1': -84.10580775218392,
'RandomForestMSE_BAG_L1': -119.08177302978171,
'CatBoost_BAG_L1': -130.46120460893425,
'ExtraTreesMSE_BAG_L1': -126.80149965568127,
'NeuralNetFastAI_BAG_L1': -135.67526066842626,
'XGBoost_BAG_L1': -245.17431441781224,
'NeuralNetTorch_BAG_L1': -139.90515963131176,
'WeightedEnsemble_L2': -84.10580775218392,
'RandomForestMSE_BAG_L2': -54.16425013512568,
'CatBoost_BAG_L2': -55.38272690001613,
'ExtraTreesMSE_BAG_L2': -54.302186838226554,
'NeuralNetFastAI_BAG_L2': -52.28097726236311,
'XGBoost_BAG_L2': -239.1839116086003,
'NeuralNetTorch_BAG_L2': -58.441239557285215,
'WeightedEnsemble_L3': -51.02515926002657},
'model_best': 'WeightedEnsemble_L3',
'model_paths': {'KNeighborsUnif_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/KNeighborsUnif_BAG_L1/',
'KNeighborsDist_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/KNeighborsDist_BAG_L1/',
'RandomForestMSE_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/RandomForestMSE_BAG_L1/',
'CatBoost_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/CatBoost_BAG_L1/',
'ExtraTreesMSE_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/ExtraTreesMSE_BAG_L1/',
'NeuralNetFastAI_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/NeuralNetFastAI_BAG_L1/',
'XGBoost_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/XGBoost_BAG_L1/',
'NeuralNetTorch_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/NeuralNetTorch_BAG_L1/',
'WeightedEnsemble_L2': 'AutogluonModels/ag-20220403_215705/models/WeightedEnsemble_L2/',
'RandomForestMSE_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/RandomForestMSE_BAG_L2/',
'CatBoost_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/CatBoost_BAG_L2/',
'ExtraTreesMSE_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/ExtraTreesMSE_BAG_L2/',
'NeuralNetFastAI_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/NeuralNetFastAI_BAG_L2/',
'XGBoost_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/XGBoost_BAG_L2/',
'NeuralNetTorch_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/NeuralNetTorch_BAG_L2/',
'WeightedEnsemble_L3': 'AutogluonModels/ag-20220403_215705/models/WeightedEnsemble_L3/'},
'model_fit_times': {'KNeighborsUnif_BAG_L1': 0.04007411003112793,
'KNeighborsDist_BAG_L1': 0.01625204086303711,
'RandomForestMSE_BAG_L1': 0.43125224113464355,
'CatBoost_BAG_L1': 192.76384949684143,
'ExtraTreesMSE_BAG_L1': 0.26133012771606445,
'NeuralNetFastAI_BAG_L1': 56.096115827560425,
'XGBoost_BAG_L1': 0.17375564575195312,
'NeuralNetTorch_BAG_L1': 142.73352360725403,
'WeightedEnsemble_L2': 0.14844107627868652,
'RandomForestMSE_BAG_L2': 1.2392151355743408,
'CatBoost_BAG_L2': 31.919593572616577,
'ExtraTreesMSE_BAG_L2': 0.3651919364929199,
'NeuralNetFastAI_BAG_L2': 57.0677490234375,
'XGBoost_BAG_L2': 0.25720643997192383,
'NeuralNetTorch_BAG_L2': 109.81508946418762,
'WeightedEnsemble_L3': 0.1198110580444336},
'model_pred_times': {'KNeighborsUnif_BAG_L1': 0.027614831924438477,
'KNeighborsDist_BAG_L1': 0.030787944793701172,
'RandomForestMSE_BAG_L1': 0.06241106986999512,
'CatBoost_BAG_L1': 0.03171730041503906,
'ExtraTreesMSE_BAG_L1': 0.06524085998535156,
'NeuralNetFastAI_BAG_L1': 0.10861992835998535,
'XGBoost_BAG_L1': 0.030724525451660156,
'NeuralNetTorch_BAG_L1': 0.13529324531555176,
'WeightedEnsemble_L2': 0.00026106834411621094,
'RandomForestMSE_BAG_L2': 0.07856607437133789,
'CatBoost_BAG_L2': 0.02326655387878418,
'ExtraTreesMSE_BAG_L2': 0.07741594314575195,
'NeuralNetFastAI_BAG_L2': 0.1224508285522461,
'XGBoost_BAG_L2': 0.0399935245513916,
'NeuralNetTorch_BAG_L2': 0.15211033821105957,
'WeightedEnsemble_L3': 0.0002548694610595703},
'num_bag_folds': 8,
'max_stack_level': 3,
'model_hyperparams': {'KNeighborsUnif_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'KNeighborsDist_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'RandomForestMSE_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'CatBoost_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'ExtraTreesMSE_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'NeuralNetFastAI_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'XGBoost_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'NeuralNetTorch_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'WeightedEnsemble_L2': {'use_orig_features': False,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'RandomForestMSE_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'CatBoost_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'ExtraTreesMSE_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'NeuralNetFastAI_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'XGBoost_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'NeuralNetTorch_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'WeightedEnsemble_L3': {'use_orig_features': False,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True}},
'leaderboard': model score_val pred_time_val fit_time \
0 WeightedEnsemble_L3 -51.025159 0.771097 451.308120
1 NeuralNetFastAI_BAG_L2 -52.280977 0.614861 449.583902
2 RandomForestMSE_BAG_L2 -54.164250 0.570976 393.755368
3 ExtraTreesMSE_BAG_L2 -54.302187 0.569826 392.881345
4 CatBoost_BAG_L2 -55.382727 0.515676 424.435747
5 NeuralNetTorch_BAG_L2 -58.441240 0.644520 502.331243
6 KNeighborsDist_BAG_L1 -84.105808 0.030788 0.016252
7 WeightedEnsemble_L2 -84.105808 0.031049 0.164693
8 KNeighborsUnif_BAG_L1 -101.506986 0.027615 0.040074
9 RandomForestMSE_BAG_L1 -119.081773 0.062411 0.431252
10 ExtraTreesMSE_BAG_L1 -126.801500 0.065241 0.261330
11 CatBoost_BAG_L1 -130.461205 0.031717 192.763849
12 NeuralNetFastAI_BAG_L1 -135.675261 0.108620 56.096116
13 NeuralNetTorch_BAG_L1 -139.905160 0.135293 142.733524
14 XGBoost_BAG_L2 -239.183912 0.532403 392.773360
15 XGBoost_BAG_L1 -245.174314 0.030725 0.173756
pred_time_val_marginal fit_time_marginal stack_level can_infer \
0 0.000255 0.119811 3 True
1 0.122451 57.067749 2 True
2 0.078566 1.239215 2 True
3 0.077416 0.365192 2 True
4 0.023267 31.919594 2 True
5 0.152110 109.815089 2 True
6 0.030788 0.016252 1 True
7 0.000261 0.148441 2 True
8 0.027615 0.040074 1 True
9 0.062411 0.431252 1 True
10 0.065241 0.261330 1 True
11 0.031717 192.763849 1 True
12 0.108620 56.096116 1 True
13 0.135293 142.733524 1 True
14 0.039994 0.257206 2 True
15 0.030725 0.173756 1 True
fit_order
0 16
1 13
2 10
3 12
4 11
5 15
6 2
7 9
8 1
9 3
10 5
11 4
12 6
13 8
14 14
15 7 }
# Output the fit summary of the training run
predictor.leaderboard(silent=True).plot(kind="bar", x="model", y="score_val")
<AxesSubplot:xlabel='model'>
predictions = predictor.predict(test)
predictions.head()
0 25.395908 1 40.721481 2 46.926262 3 48.073616 4 49.973679 Name: count, dtype: float32
# Describe the `predictions` series to see if there are any negative values
predictions.describe()
count 6493.000000 mean 99.825455 std 88.543869 min 1.512626 25% 20.131336 50% 67.336700 75% 169.070740 max 356.121338 Name: count, dtype: float64
# How many negative values do we have?
predictions.lt(0).sum()
0
# Set them to zero
predictions.clip(lower=0)
0 25.395908
1 40.721481
2 46.926262
3 48.073616
4 49.973679
...
6488 170.985306
6489 170.989395
6490 157.917023
6491 154.561798
6492 148.553497
Name: count, Length: 6493, dtype: float32
submission["count"] = predictions
# Round count column values
submission["count"] = submission["count"].apply(lambda x: round(x))
submission.to_csv("submission.csv", index=False)
submission.head()
| datetime | count | |
|---|---|---|
| 0 | 2011-01-20 00:00:00 | 25 |
| 1 | 2011-01-20 01:00:00 | 41 |
| 2 | 2011-01-20 02:00:00 | 47 |
| 3 | 2011-01-20 03:00:00 | 48 |
| 4 | 2011-01-20 04:00:00 | 50 |
!kaggle competitions submit -c bike-sharing-demand -f submission.csv -m "first raw submission"
100%|████████████████████████████████████████| 148k/148k [00:03<00:00, 47.1kB/s] Successfully submitted to Bike Sharing Demand
My Submissions¶!kaggle competitions submissions -c bike-sharing-demand | tail -n +1 | head -n 6
fileName date description status publicScore privateScore -------------- ------------------- -------------------- -------- ----------- ------------ submission.csv 2022-04-03 22:23:11 first raw submission complete 1.80656 1.80656
1.80656¶from pandas_profiling import ProfileReport
profile = ProfileReport(train, title="Pandas Profiling Report")
profile
# Create a histogram of all features to show the distribution of each one relative to the data.
# This is part of the exploritory data analysis
train = train.drop(['casual','registered'], axis=1)
train.hist(figsize=(15, 10));
# time series plot for demand
train.plot(x="datetime", y="count", figsize=(15,6))
<AxesSubplot:xlabel='datetime'>
# separate out the datetime into hour, day, month parts
train["year"] = train["datetime"].dt.year
train["month"] = train["datetime"].dt.month
train["day"] = train["datetime"].dt.day
train["day_of_week"] = train["datetime"].dt.day_of_week
train["hour"] = train["datetime"].dt.hour
# add same teatures in test set
test["year"] = test["datetime"].dt.year
test["month"] = test["datetime"].dt.month
test["day"] = test["datetime"].dt.day
test["day_of_week"] = test["datetime"].dt.day_of_week
test["hour"] = test["datetime"].dt.hour
train.head()
| datetime | season | holiday | workingday | weather | temp | atemp | humidity | windspeed | count | year | month | day | day_of_week | hour | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2011-01-01 00:00:00 | 1 | 0 | 0 | 1 | 9.84 | 14.395 | 81 | 0.0 | 16 | 2011 | 1 | 1 | 5 | 0 |
| 1 | 2011-01-01 01:00:00 | 1 | 0 | 0 | 1 | 9.02 | 13.635 | 80 | 0.0 | 40 | 2011 | 1 | 1 | 5 | 1 |
| 2 | 2011-01-01 02:00:00 | 1 | 0 | 0 | 1 | 9.02 | 13.635 | 80 | 0.0 | 32 | 2011 | 1 | 1 | 5 | 2 |
| 3 | 2011-01-01 03:00:00 | 1 | 0 | 0 | 1 | 9.84 | 14.395 | 75 | 0.0 | 13 | 2011 | 1 | 1 | 5 | 3 |
| 4 | 2011-01-01 04:00:00 | 1 | 0 | 0 | 1 | 9.84 | 14.395 | 75 | 0.0 | 1 | 2011 | 1 | 1 | 5 | 4 |
# time series plot for a paticular month: 2011-4
train[ (train["year"]==2011) & (train["month"]==4) ].plot(x="datetime", y="count", figsize=(15,6))
<AxesSubplot:xlabel='datetime'>
# time series plot for a paticular week: 2011-4-1 - 2011-4-8
train[
(train["datetime"]>='2011-4-1') &
(train["datetime"]<='2011-4-8')
].plot(x="datetime", y="count", figsize=(15,6))
<AxesSubplot:xlabel='datetime'>
# plot for all working days
plt.figure(figsize=(15,6))
sns.barplot(x="hour",y="count",data=train[train["workingday"]==1], color=sns.color_palette()[0], ci=None)
<AxesSubplot:xlabel='hour', ylabel='count'>
# plot for all weekend
plt.figure(figsize=(15,6))
sns.barplot(x="hour",y="count",data=train[train["workingday"]==0], color=sns.color_palette()[0], ci=None)
<AxesSubplot:xlabel='hour', ylabel='count'>
# compute the correlation between "temp" and "atemp"
train[["temp", "atemp"]].corr()
| temp | atemp | |
|---|---|---|
| temp | 1.000000 | 0.984948 |
| atemp | 0.984948 | 1.000000 |
# we choose temp rather than atemp becasue the distribution "feel-like" temp has more bias
# plot for temp
plt.figure(figsize=(15,6))
sns.barplot(x="temp",y="count", data=train, color=sns.color_palette()[0], ci=None)
plt.xticks(rotation = 90);
# create new feature: temp_category
def get_temp_category(temp):
if temp <= 13:
return 0 # low temp
elif (temp > 13) and (temp <= 28 ):
return 1 # mild temp
elif (temp > 28):
return 2 # hot temp
train['temp_category'] = train['temp'].apply(get_temp_category)
test['temp_category'] = test['temp'].apply(get_temp_category)
# plot for humidity
plt.figure(figsize=(15,6))
sns.barplot(x="humidity", y="count", data=train, color=sns.color_palette()[0], ci=None)
plt.xticks(rotation = 90);
# create new feature: humidity_category
def get_humid_category(humid):
if humid <= 15:
return 0 # low temp
elif (humid > 15) and (humid <= 46 ):
return 1 # mild temp
elif (humid > 46) and (humid < 75 ):
return 2 # hot temp
else:
return 3
train['humid_category'] = train['humidity'].apply(get_humid_category)
test['humid_category'] = test['humidity'].apply(get_humid_category)
# the distribution of windspeed shows there are outliers
sns.boxplot(data=train, y="windspeed",orient="v")
<AxesSubplot:ylabel='windspeed'>
train = train[train["windspeed"]-train["windspeed"].mean() <=(3*train["windspeed"].std())]
# plot for windspeed
plt.figure(figsize=(15,6))
sns.barplot(x="windspeed", y="count", data=train, color=sns.color_palette()[0], ci=None)
plt.xticks(rotation = 90);
# create new feature: wind_category
def get_wind_category(wind):
if wind <= 10:
return 0 # low wind
elif (wind > 10) and (wind <= 20 ):
return 1 # mild wind
elif (wind > 20):
return 2 # high temp
train['wind_category'] = train['windspeed'].apply(get_wind_category)
test['wind_category'] = test['windspeed'].apply(get_wind_category)
# drop unnecessary features
train.drop(['datetime', 'temp', 'windspeed', 'humidity', 'atemp'], axis=1, inplace=True)
test.drop(['datetime', 'temp', 'windspeed', 'humidity', 'atemp'], axis=1, inplace=True)
# plot correlations between features
plt.figure(figsize=(10, 8), dpi=80)
sns.heatmap(train.drop(['count'], axis=1).corr(), annot=True)
plt.show()
# removing season column as it's having very high correlation with month
train.drop(['season', 'day_of_week'], axis=1, inplace=True)
test.drop(['season', 'day_of_week'], axis=1, inplace=True)
train["weather"] = train["weather"].astype('category')
train["temp_category"] = train["temp_category"].astype('category')
train["wind_category"] = train["wind_category"].astype('category')
train["humid_category"] = train["humid_category"].astype('category')
train["holiday"] = train["holiday"].astype('category')
train["workingday"] = train["workingday"].astype('category')
train["hour"] = train["hour"].astype('category')
train["month"] = train["month"].astype('category')
test["weather"] = test["weather"].astype('category')
test["temp_category"] = test["temp_category"].astype('category')
test["wind_category"] = test["wind_category"].astype('category')
test["humid_category"] = test["humid_category"].astype('category')
test["holiday"] = test["holiday"].astype('category')
test["workingday"] = test["workingday"].astype('category')
test["hour"] = test["hour"].astype('category')
test["month"] = test["month"].astype('category')
train.info()
<class 'pandas.core.frame.DataFrame'> Int64Index: 10819 entries, 0 to 10885 Data columns (total 11 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 holiday 10819 non-null category 1 workingday 10819 non-null category 2 weather 10819 non-null category 3 count 10819 non-null int64 4 year 10819 non-null int64 5 month 10819 non-null category 6 day 10819 non-null int64 7 hour 10819 non-null category 8 temp_category 10819 non-null category 9 humid_category 10819 non-null category 10 wind_category 10819 non-null category dtypes: category(8), int64(3) memory usage: 682.7 KB
# View are new feature
train.head()
| holiday | workingday | weather | count | year | month | day | hour | temp_category | humid_category | wind_category | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 1 | 16 | 2011 | 1 | 1 | 0 | 0 | 3 | 0 |
| 1 | 0 | 0 | 1 | 40 | 2011 | 1 | 1 | 1 | 0 | 3 | 0 |
| 2 | 0 | 0 | 1 | 32 | 2011 | 1 | 1 | 2 | 0 | 3 | 0 |
| 3 | 0 | 0 | 1 | 13 | 2011 | 1 | 1 | 3 | 0 | 3 | 0 |
| 4 | 0 | 0 | 1 | 1 | 2011 | 1 | 1 | 4 | 0 | 3 | 0 |
# View histogram of all features again now with the hour feature
train.hist(figsize=(15,10));
predictor_new_features = TabularPredictor(
label="count",
problem_type="regression",
eval_metric="root_mean_squared_error"
).fit(
train_data=train,
time_limit=600,
presets="best_quality"
)
No path specified. Models will be saved in: "AutogluonModels/ag-20220404_171935/"
Presets specified: ['best_quality']
Beginning AutoGluon training ... Time limit = 600s
AutoGluon will save models to "AutogluonModels/ag-20220404_171935/"
AutoGluon Version: 0.4.0
Python Version: 3.9.7
Operating System: Darwin
Train Data Rows: 10819
Train Data Columns: 10
Label Column: count
Preprocessing data ...
Using Feature Generators to preprocess the data ...
Fitting AutoMLPipelineFeatureGenerator...
Available Memory: 399.34 MB
Train Data (Original) Memory Usage: 0.26 MB (0.1% of available memory)
Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
Stage 1 Generators:
Fitting AsTypeFeatureGenerator...
Note: Converting 3 features to boolean dtype as they only contain 2 unique values.
Stage 2 Generators:
Fitting FillNaFeatureGenerator...
Stage 3 Generators:
Fitting IdentityFeatureGenerator...
Fitting CategoryFeatureGenerator...
Fitting CategoryMemoryMinimizeFeatureGenerator...
Stage 4 Generators:
Fitting DropUniqueFeatureGenerator...
Types of features in original data (raw dtype, special dtypes):
('category', []) : 8 | ['holiday', 'workingday', 'weather', 'month', 'hour', ...]
('int', []) : 2 | ['year', 'day']
Types of features in processed data (raw dtype, special dtypes):
('category', []) : 6 | ['weather', 'month', 'hour', 'temp_category', 'humid_category', ...]
('int', []) : 1 | ['day']
('int', ['bool']) : 3 | ['holiday', 'workingday', 'year']
0.1s = Fit runtime
10 features in original data used to generate 10 features in processed data.
Train Data (Processed) Memory Usage: 0.19 MB (0.0% of available memory)
Data preprocessing and feature engineering runtime = 0.07s ...
AutoGluon will gauge predictive performance using evaluation metric: 'root_mean_squared_error'
To change this, specify the eval_metric parameter of Predictor()
AutoGluon will fit 2 stack levels (L1 to L2) ...
Fitting 11 L1 models ...
Fitting model: KNeighborsUnif_BAG_L1 ... Training model for up to 399.85s of the 599.93s of remaining time.
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
-246.8409 = Validation score (root_mean_squared_error)
2.61s = Training runtime
0.04s = Validation runtime
Fitting model: KNeighborsDist_BAG_L1 ... Training model for up to 397.16s of the 597.23s of remaining time.
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
-246.8409 = Validation score (root_mean_squared_error)
0.01s = Training runtime
0.05s = Validation runtime
Fitting model: LightGBMXT_BAG_L1 ... Training model for up to 397.09s of the 597.16s of remaining time.
Will use sequential fold fitting strategy because Darwin OS does not yet support parallel folding.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBMXT_BAG_L1 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Fitting model: LightGBM_BAG_L1 ... Training model for up to 396.47s of the 596.55s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBM_BAG_L1 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Fitting model: RandomForestMSE_BAG_L1 ... Training model for up to 396.46s of the 596.53s of remaining time.
Warning: Reducing model 'n_estimators' from 300 -> 65 due to low memory. Expected memory usage reduced from 68.23% -> 15.0% of available memory...
-44.4183 = Validation score (root_mean_squared_error)
0.28s = Training runtime
0.06s = Validation runtime
Fitting model: CatBoost_BAG_L1 ... Training model for up to 396.07s of the 596.14s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Ran out of time, early stopping on iteration 6141.
Ran out of time, early stopping on iteration 6324.
Ran out of time, early stopping on iteration 6676.
Ran out of time, early stopping on iteration 7916.
Ran out of time, early stopping on iteration 8392.
Ran out of time, early stopping on iteration 9546.
-45.5806 = Validation score (root_mean_squared_error)
377.48s = Training runtime
0.1s = Validation runtime
Fitting model: ExtraTreesMSE_BAG_L1 ... Training model for up to 18.26s of the 218.34s of remaining time.
Warning: Reducing model 'n_estimators' from 300 -> 64 due to low memory. Expected memory usage reduced from 70.19% -> 15.0% of available memory...
-45.6605 = Validation score (root_mean_squared_error)
0.21s = Training runtime
0.07s = Validation runtime
Fitting model: NeuralNetFastAI_BAG_L1 ... Training model for up to 17.92s of the 218.0s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Ran out of time, stopping training early. (Stopping on epoch 3)
Time limit exceeded... Skipping NeuralNetFastAI_BAG_L1.
Fitting model: XGBoost_BAG_L1 ... Training model for up to 15.33s of the 215.4s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 338 MB
Estimated XGB model size: 3 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 330 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 325 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 325 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 325 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 323 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 324 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 325 MB
Estimated XGB model size: 0 MB
-243.6172 = Validation score (root_mean_squared_error)
0.4s = Training runtime
0.05s = Validation runtime
Fitting model: NeuralNetTorch_BAG_L1 ... Training model for up to 14.86s of the 214.93s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Ran out of time, stopping training early. (Stopping on epoch 3)
Ran out of time, stopping training early. (Stopping on epoch 3)
Ran out of time, stopping training early. (Stopping on epoch 4)
Ran out of time, stopping training early. (Stopping on epoch 4)
Ran out of time, stopping training early. (Stopping on epoch 5)
-102.5411 = Validation score (root_mean_squared_error)
13.79s = Training runtime
0.19s = Validation runtime
Fitting model: LightGBMLarge_BAG_L1 ... Training model for up to 0.85s of the 200.93s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBMLarge_BAG_L1 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Completed 1/20 k-fold bagging repeats ...
Fitting model: WeightedEnsemble_L2 ... Training model for up to 360.0s of the 200.9s of remaining time.
-41.9413 = Validation score (root_mean_squared_error)
0.14s = Training runtime
0.0s = Validation runtime
Fitting 9 L2 models ...
Fitting model: LightGBMXT_BAG_L2 ... Training model for up to 200.75s of the 200.75s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBMXT_BAG_L2 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Fitting model: LightGBM_BAG_L2 ... Training model for up to 200.73s of the 200.73s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBM_BAG_L2 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Fitting model: RandomForestMSE_BAG_L2 ... Training model for up to 200.71s of the 200.71s of remaining time.
Warning: Reducing model 'n_estimators' from 300 -> 66 due to low memory. Expected memory usage reduced from 67.91% -> 15.0% of available memory...
-43.7209 = Validation score (root_mean_squared_error)
0.77s = Training runtime
0.07s = Validation runtime
Fitting model: CatBoost_BAG_L2 ... Training model for up to 199.8s of the 199.79s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
-42.3145 = Validation score (root_mean_squared_error)
20.97s = Training runtime
0.04s = Validation runtime
Fitting model: ExtraTreesMSE_BAG_L2 ... Training model for up to 178.75s of the 178.75s of remaining time.
Warning: Reducing model 'n_estimators' from 300 -> 64 due to low memory. Expected memory usage reduced from 69.86% -> 15.0% of available memory...
-42.7483 = Validation score (root_mean_squared_error)
0.28s = Training runtime
0.07s = Validation runtime
Fitting model: NeuralNetFastAI_BAG_L2 ... Training model for up to 178.35s of the 178.34s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
No improvement since epoch 4: early stopping
No improvement since epoch 6: early stopping
No improvement since epoch 3: early stopping
No improvement since epoch 4: early stopping
No improvement since epoch 2: early stopping
-42.5218 = Validation score (root_mean_squared_error)
58.3s = Training runtime
0.16s = Validation runtime
Fitting model: XGBoost_BAG_L2 ... Training model for up to 119.8s of the 119.79s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 318 MB
Estimated XGB model size: 21 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 318 MB
Estimated XGB model size: 9 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 319 MB
Estimated XGB model size: 7 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 317 MB
Estimated XGB model size: 4 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 316 MB
Estimated XGB model size: 7 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 321 MB
Estimated XGB model size: 3 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 322 MB
Estimated XGB model size: 2 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 322 MB
Estimated XGB model size: 0 MB
-238.0238 = Validation score (root_mean_squared_error)
0.5s = Training runtime
0.05s = Validation runtime
Fitting model: NeuralNetTorch_BAG_L2 ... Training model for up to 119.22s of the 119.22s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Ran out of time, stopping training early. (Stopping on epoch 26)
Ran out of time, stopping training early. (Stopping on epoch 27)
Ran out of time, stopping training early. (Stopping on epoch 28)
Ran out of time, stopping training early. (Stopping on epoch 29)
Ran out of time, stopping training early. (Stopping on epoch 31)
Ran out of time, stopping training early. (Stopping on epoch 33)
Ran out of time, stopping training early. (Stopping on epoch 36)
Ran out of time, stopping training early. (Stopping on epoch 45)
-43.8899 = Validation score (root_mean_squared_error)
113.69s = Training runtime
0.23s = Validation runtime
Fitting model: LightGBMLarge_BAG_L2 ... Training model for up to 5.27s of the 5.26s of remaining time.
Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Exception caused LightGBMLarge_BAG_L2 to fail during training (ImportError)... Skipping this model.
`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Completed 1/20 k-fold bagging repeats ...
Fitting model: WeightedEnsemble_L3 ... Training model for up to 360.0s of the 5.21s of remaining time.
-41.6628 = Validation score (root_mean_squared_error)
0.12s = Training runtime
0.0s = Validation runtime
AutoGluon training complete, total runtime = 594.92s ... Best model: "WeightedEnsemble_L3"
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("AutogluonModels/ag-20220404_171935/")
predictor_new_features.fit_summary()
*** Summary of fit() ***
Estimated performance of each model:
model score_val pred_time_val fit_time pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order
0 WeightedEnsemble_L3 -41.662787 1.137516 588.921629 0.000282 0.122048 3 True 15
1 WeightedEnsemble_L2 -41.941318 0.232135 378.103965 0.000266 0.138094 2 True 8
2 CatBoost_BAG_L2 -42.314486 0.601931 415.760942 0.044700 20.974016 2 True 10
3 NeuralNetFastAI_BAG_L2 -42.521774 0.715616 453.085581 0.158386 58.298656 2 True 12
4 ExtraTreesMSE_BAG_L2 -42.748345 0.627164 395.070385 0.069933 0.283460 2 True 11
5 RandomForestMSE_BAG_L2 -43.720860 0.630273 395.556621 0.073042 0.769696 2 True 9
6 NeuralNetTorch_BAG_L2 -43.889909 0.791174 508.473753 0.233943 113.686828 2 True 14
7 RandomForestMSE_BAG_L1 -44.418272 0.064314 0.276485 0.064314 0.276485 1 True 3
8 CatBoost_BAG_L1 -45.580553 0.101993 377.483847 0.101993 377.483847 1 True 4
9 ExtraTreesMSE_BAG_L1 -45.660535 0.065563 0.205539 0.065563 0.205539 1 True 5
10 NeuralNetTorch_BAG_L1 -102.541134 0.185994 13.794680 0.185994 13.794680 1 True 7
11 XGBoost_BAG_L2 -238.023805 0.609681 395.291189 0.052450 0.504264 2 True 13
12 XGBoost_BAG_L1 -243.617185 0.049562 0.403421 0.049562 0.403421 1 True 6
13 KNeighborsUnif_BAG_L1 -246.840883 0.042858 2.608711 0.042858 2.608711 1 True 1
14 KNeighborsDist_BAG_L1 -246.840883 0.046947 0.014242 0.046947 0.014242 1 True 2
Number of models trained: 15
Types of models trained:
{'WeightedEnsembleModel', 'StackerEnsembleModel_KNN', 'StackerEnsembleModel_XT', 'StackerEnsembleModel_CatBoost', 'StackerEnsembleModel_XGBoost', 'StackerEnsembleModel_NNFastAiTabular', 'StackerEnsembleModel_TabularNeuralNetTorch', 'StackerEnsembleModel_RF'}
Bagging used: True (with 8 folds)
Multi-layer stack-ensembling used: True (with 3 levels)
Feature Metadata (Processed):
(raw dtype, special dtypes):
('category', []) : 6 | ['weather', 'month', 'hour', 'temp_category', 'humid_category', ...]
('int', []) : 1 | ['day']
('int', ['bool']) : 3 | ['holiday', 'workingday', 'year']
Plot summary of models saved to file: AutogluonModels/ag-20220404_171935/SummaryOfModels.html
*** End of fit() summary ***
{'model_types': {'KNeighborsUnif_BAG_L1': 'StackerEnsembleModel_KNN',
'KNeighborsDist_BAG_L1': 'StackerEnsembleModel_KNN',
'RandomForestMSE_BAG_L1': 'StackerEnsembleModel_RF',
'CatBoost_BAG_L1': 'StackerEnsembleModel_CatBoost',
'ExtraTreesMSE_BAG_L1': 'StackerEnsembleModel_XT',
'XGBoost_BAG_L1': 'StackerEnsembleModel_XGBoost',
'NeuralNetTorch_BAG_L1': 'StackerEnsembleModel_TabularNeuralNetTorch',
'WeightedEnsemble_L2': 'WeightedEnsembleModel',
'RandomForestMSE_BAG_L2': 'StackerEnsembleModel_RF',
'CatBoost_BAG_L2': 'StackerEnsembleModel_CatBoost',
'ExtraTreesMSE_BAG_L2': 'StackerEnsembleModel_XT',
'NeuralNetFastAI_BAG_L2': 'StackerEnsembleModel_NNFastAiTabular',
'XGBoost_BAG_L2': 'StackerEnsembleModel_XGBoost',
'NeuralNetTorch_BAG_L2': 'StackerEnsembleModel_TabularNeuralNetTorch',
'WeightedEnsemble_L3': 'WeightedEnsembleModel'},
'model_performance': {'KNeighborsUnif_BAG_L1': -246.84088280283666,
'KNeighborsDist_BAG_L1': -246.84088280283666,
'RandomForestMSE_BAG_L1': -44.41827240398544,
'CatBoost_BAG_L1': -45.58055346129427,
'ExtraTreesMSE_BAG_L1': -45.66053480825233,
'XGBoost_BAG_L1': -243.6171849564755,
'NeuralNetTorch_BAG_L1': -102.54113388241504,
'WeightedEnsemble_L2': -41.94131756111479,
'RandomForestMSE_BAG_L2': -43.720860491299774,
'CatBoost_BAG_L2': -42.314486219830165,
'ExtraTreesMSE_BAG_L2': -42.74834515629869,
'NeuralNetFastAI_BAG_L2': -42.52177351470511,
'XGBoost_BAG_L2': -238.0238054930029,
'NeuralNetTorch_BAG_L2': -43.88990930176027,
'WeightedEnsemble_L3': -41.662786751754},
'model_best': 'WeightedEnsemble_L3',
'model_paths': {'KNeighborsUnif_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/KNeighborsUnif_BAG_L1/',
'KNeighborsDist_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/KNeighborsDist_BAG_L1/',
'RandomForestMSE_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/RandomForestMSE_BAG_L1/',
'CatBoost_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/CatBoost_BAG_L1/',
'ExtraTreesMSE_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/ExtraTreesMSE_BAG_L1/',
'XGBoost_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/XGBoost_BAG_L1/',
'NeuralNetTorch_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/NeuralNetTorch_BAG_L1/',
'WeightedEnsemble_L2': 'AutogluonModels/ag-20220404_171935/models/WeightedEnsemble_L2/',
'RandomForestMSE_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/RandomForestMSE_BAG_L2/',
'CatBoost_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/CatBoost_BAG_L2/',
'ExtraTreesMSE_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/ExtraTreesMSE_BAG_L2/',
'NeuralNetFastAI_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/NeuralNetFastAI_BAG_L2/',
'XGBoost_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/XGBoost_BAG_L2/',
'NeuralNetTorch_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/NeuralNetTorch_BAG_L2/',
'WeightedEnsemble_L3': 'AutogluonModels/ag-20220404_171935/models/WeightedEnsemble_L3/'},
'model_fit_times': {'KNeighborsUnif_BAG_L1': 2.608711004257202,
'KNeighborsDist_BAG_L1': 0.014241933822631836,
'RandomForestMSE_BAG_L1': 0.27648472785949707,
'CatBoost_BAG_L1': 377.48384737968445,
'ExtraTreesMSE_BAG_L1': 0.20553922653198242,
'XGBoost_BAG_L1': 0.40342116355895996,
'NeuralNetTorch_BAG_L1': 13.794679880142212,
'WeightedEnsemble_L2': 0.1380939483642578,
'RandomForestMSE_BAG_L2': 0.7696959972381592,
'CatBoost_BAG_L2': 20.974016427993774,
'ExtraTreesMSE_BAG_L2': 0.2834599018096924,
'NeuralNetFastAI_BAG_L2': 58.29865550994873,
'XGBoost_BAG_L2': 0.5042638778686523,
'NeuralNetTorch_BAG_L2': 113.68682765960693,
'WeightedEnsemble_L3': 0.12204790115356445},
'model_pred_times': {'KNeighborsUnif_BAG_L1': 0.042858123779296875,
'KNeighborsDist_BAG_L1': 0.04694700241088867,
'RandomForestMSE_BAG_L1': 0.06431388854980469,
'CatBoost_BAG_L1': 0.10199284553527832,
'ExtraTreesMSE_BAG_L1': 0.06556272506713867,
'XGBoost_BAG_L1': 0.04956173896789551,
'NeuralNetTorch_BAG_L1': 0.18599438667297363,
'WeightedEnsemble_L2': 0.0002658367156982422,
'RandomForestMSE_BAG_L2': 0.07304215431213379,
'CatBoost_BAG_L2': 0.044699907302856445,
'ExtraTreesMSE_BAG_L2': 0.06993293762207031,
'NeuralNetFastAI_BAG_L2': 0.1583855152130127,
'XGBoost_BAG_L2': 0.05245041847229004,
'NeuralNetTorch_BAG_L2': 0.23394322395324707,
'WeightedEnsemble_L3': 0.0002818107604980469},
'num_bag_folds': 8,
'max_stack_level': 3,
'model_hyperparams': {'KNeighborsUnif_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'KNeighborsDist_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'RandomForestMSE_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'CatBoost_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'ExtraTreesMSE_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'XGBoost_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'NeuralNetTorch_BAG_L1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'WeightedEnsemble_L2': {'use_orig_features': False,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'RandomForestMSE_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'CatBoost_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'ExtraTreesMSE_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'NeuralNetFastAI_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'XGBoost_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'NeuralNetTorch_BAG_L2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'WeightedEnsemble_L3': {'use_orig_features': False,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True}},
'leaderboard': model score_val pred_time_val fit_time \
0 WeightedEnsemble_L3 -41.662787 1.137516 588.921629
1 WeightedEnsemble_L2 -41.941318 0.232135 378.103965
2 CatBoost_BAG_L2 -42.314486 0.601931 415.760942
3 NeuralNetFastAI_BAG_L2 -42.521774 0.715616 453.085581
4 ExtraTreesMSE_BAG_L2 -42.748345 0.627164 395.070385
5 RandomForestMSE_BAG_L2 -43.720860 0.630273 395.556621
6 NeuralNetTorch_BAG_L2 -43.889909 0.791174 508.473753
7 RandomForestMSE_BAG_L1 -44.418272 0.064314 0.276485
8 CatBoost_BAG_L1 -45.580553 0.101993 377.483847
9 ExtraTreesMSE_BAG_L1 -45.660535 0.065563 0.205539
10 NeuralNetTorch_BAG_L1 -102.541134 0.185994 13.794680
11 XGBoost_BAG_L2 -238.023805 0.609681 395.291189
12 XGBoost_BAG_L1 -243.617185 0.049562 0.403421
13 KNeighborsUnif_BAG_L1 -246.840883 0.042858 2.608711
14 KNeighborsDist_BAG_L1 -246.840883 0.046947 0.014242
pred_time_val_marginal fit_time_marginal stack_level can_infer \
0 0.000282 0.122048 3 True
1 0.000266 0.138094 2 True
2 0.044700 20.974016 2 True
3 0.158386 58.298656 2 True
4 0.069933 0.283460 2 True
5 0.073042 0.769696 2 True
6 0.233943 113.686828 2 True
7 0.064314 0.276485 1 True
8 0.101993 377.483847 1 True
9 0.065563 0.205539 1 True
10 0.185994 13.794680 1 True
11 0.052450 0.504264 2 True
12 0.049562 0.403421 1 True
13 0.042858 2.608711 1 True
14 0.046947 0.014242 1 True
fit_order
0 15
1 8
2 10
3 12
4 11
5 9
6 14
7 3
8 4
9 5
10 7
11 13
12 6
13 1
14 2 }
# Output the fit summary of the training run
predictor_new_features.leaderboard(silent=True).plot(kind="bar", x="model", y="score_val")
<AxesSubplot:xlabel='model'>
# Remember to set all negative values to zero
predictions = predictor_new_features.predict(test)
predictions = predictions.clip(lower=0)
# Same submitting predictions
submission_new_features = pd.read_csv('data/sampleSubmission.csv', parse_dates=['datetime'])
submission_new_features["count"] = predictions
submission_new_features['count'] = submission_new_features['count'].apply(lambda x: round(x))
submission_new_features.to_csv("submission_new_features.csv", index=False)
submission_new_features.head()
| datetime | count | |
|---|---|---|
| 0 | 2011-01-20 00:00:00 | 15 |
| 1 | 2011-01-20 01:00:00 | 6 |
| 2 | 2011-01-20 02:00:00 | 3 |
| 3 | 2011-01-20 03:00:00 | 4 |
| 4 | 2011-01-20 04:00:00 | 3 |
!kaggle competitions submit -c bike-sharing-demand -f submission_new_features.csv -m "new features"
100%|████████████████████████████████████████| 149k/149k [00:02<00:00, 72.5kB/s] Successfully submitted to Bike Sharing Demand
!kaggle competitions submissions -c bike-sharing-demand | tail -n +1 | head -n 6
fileName date description status publicScore privateScore --------------------------- ------------------- -------------------- -------- ----------- ------------ submission_new_features.csv 2022-04-04 17:33:48 new features complete 0.50239 0.50239 submission.csv 2022-04-03 22:23:11 first raw submission complete 1.80656 1.80656
0.50239¶hyperparameter and hyperparameter_tune_kwargs arguments.import autogluon.core as ag
nn_options = {
'num_epochs': 10,
'learning_rate': ag.space.Real(1e-4, 1e-2, default=5e-4, log=True),
'activation': ag.space.Categorical('relu', 'softrelu', 'tanh'),
'layers': ag.space.Categorical([100],[1000],[200,100],[300,200,100]),
'dropout_prob': ag.space.Real(0.0, 0.5, default=0.1),
}
gbm_options = {
'num_boost_round': 100,
'num_leaves': ag.space.Int(lower=26, upper=66, default=36)
}
hyperparameters = {
'GBM': gbm_options,
'CAT': {'iterations': 10000},
'RF': {'n_estimators': 300},
'XT': {'n_estimators': 300}
# 'NN': nn_options
}
hyperparameter_tune_kwargs = {
'scheduler': 'local',
'searcher': 'random'
}
predictor_new_hpo = TabularPredictor(
label="count",
verbosity=0,
eval_metric='root_mean_squared_error'
).fit(
train_data=train,
time_limit=600,
presets="best_quality",
hyperparameters=hyperparameters,
hyperparameter_tune_kwargs=hyperparameter_tune_kwargs
)
predictor_new_hpo.fit_summary()
*** Summary of fit() ***
Estimated performance of each model:
model score_val pred_time_val fit_time pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order
0 CatBoost_BAG_L2/T1 -42.143245 0.140617 3.158673 0.006835 2.712248 2 True 6
1 WeightedEnsemble_L2 -43.369714 0.134118 0.521992 0.000336 0.075567 2 True 4
2 WeightedEnsemble_L3 -43.407207 0.437027 1016.738522 0.000264 0.075816 3 True 9
3 CatBoost_BAG_L2/T2 -43.751974 0.262332 1015.665507 0.128550 1015.219082 2 True 7
4 RandomForest_BAG_L1/T1 -44.340259 0.065923 0.276719 0.065923 0.276719 1 True 1
5 ExtraTrees_BAG_L2/T1 -44.938209 0.238166 0.802502 0.104384 0.356077 2 True 8
6 RandomForest_BAG_L2/T1 -45.322988 0.203829 1.087548 0.070047 0.641123 2 True 5
7 CatBoost_BAG_L1/T1 -45.454415 0.105544 505.513521 0.105544 505.513521 1 True 2
8 ExtraTrees_BAG_L1/T1 -45.547377 0.067859 0.169706 0.067859 0.169706 1 True 3
Number of models trained: 9
Types of models trained:
{'StackerEnsembleModel_XT', 'WeightedEnsembleModel', 'StackerEnsembleModel_RF', 'StackerEnsembleModel_CatBoost'}
Bagging used: True (with 8 folds)
Multi-layer stack-ensembling used: True (with 3 levels)
Feature Metadata (Processed):
(raw dtype, special dtypes):
('category', []) : 6 | ['weather', 'month', 'hour', 'temp_category', 'humid_category', ...]
('int', []) : 1 | ['day']
('int', ['bool']) : 3 | ['holiday', 'workingday', 'year']
Plot summary of models saved to file: AutogluonModels/ag-20220404_192217/SummaryOfModels.html
*** End of fit() summary ***
{'model_types': {'RandomForest_BAG_L1/T1': 'StackerEnsembleModel_RF',
'CatBoost_BAG_L1/T1': 'StackerEnsembleModel_CatBoost',
'ExtraTrees_BAG_L1/T1': 'StackerEnsembleModel_XT',
'WeightedEnsemble_L2': 'WeightedEnsembleModel',
'RandomForest_BAG_L2/T1': 'StackerEnsembleModel_RF',
'CatBoost_BAG_L2/T1': 'StackerEnsembleModel_CatBoost',
'CatBoost_BAG_L2/T2': 'StackerEnsembleModel_CatBoost',
'ExtraTrees_BAG_L2/T1': 'StackerEnsembleModel_XT',
'WeightedEnsemble_L3': 'WeightedEnsembleModel'},
'model_performance': {'RandomForest_BAG_L1/T1': -44.34025936357441,
'CatBoost_BAG_L1/T1': -45.454415170852016,
'ExtraTrees_BAG_L1/T1': -45.54737689595722,
'WeightedEnsemble_L2': -43.369714134052515,
'RandomForest_BAG_L2/T1': -45.3229883681712,
'CatBoost_BAG_L2/T1': -42.143245433872984,
'CatBoost_BAG_L2/T2': -43.75197366935809,
'ExtraTrees_BAG_L2/T1': -44.93820863029768,
'WeightedEnsemble_L3': -43.4072066103833},
'model_best': 'WeightedEnsemble_L2',
'model_paths': {'RandomForest_BAG_L1/T1': 'AutogluonModels/ag-20220404_192217/models/RandomForest_BAG_L1/T1/',
'CatBoost_BAG_L1/T1': 'AutogluonModels/ag-20220404_192217/models/CatBoost_BAG_L1/T1/',
'ExtraTrees_BAG_L1/T1': 'AutogluonModels/ag-20220404_192217/models/ExtraTrees_BAG_L1/T1/',
'WeightedEnsemble_L2': 'AutogluonModels/ag-20220404_192217/models/WeightedEnsemble_L2/',
'RandomForest_BAG_L2/T1': 'AutogluonModels/ag-20220404_192217/models/RandomForest_BAG_L2/T1/',
'CatBoost_BAG_L2/T1': 'AutogluonModels/ag-20220404_192217/models/CatBoost_BAG_L2/T1/',
'CatBoost_BAG_L2/T2': 'AutogluonModels/ag-20220404_192217/models/CatBoost_BAG_L2/T2/',
'ExtraTrees_BAG_L2/T1': 'AutogluonModels/ag-20220404_192217/models/ExtraTrees_BAG_L2/T1/',
'WeightedEnsemble_L3': 'AutogluonModels/ag-20220404_192217/models/WeightedEnsemble_L3/'},
'model_fit_times': {'RandomForest_BAG_L1/T1': 0.2767190933227539,
'CatBoost_BAG_L1/T1': 505.5135214328766,
'ExtraTrees_BAG_L1/T1': 0.16970586776733398,
'WeightedEnsemble_L2': 0.07556676864624023,
'RandomForest_BAG_L2/T1': 0.6411228179931641,
'CatBoost_BAG_L2/T1': 2.7122480869293213,
'CatBoost_BAG_L2/T2': 1015.2190816402435,
'ExtraTrees_BAG_L2/T1': 0.356076717376709,
'WeightedEnsemble_L3': 0.07581591606140137},
'model_pred_times': {'RandomForest_BAG_L1/T1': 0.06592273712158203,
'CatBoost_BAG_L1/T1': 0.10554361343383789,
'ExtraTrees_BAG_L1/T1': 0.06785917282104492,
'WeightedEnsemble_L2': 0.00033593177795410156,
'RandomForest_BAG_L2/T1': 0.07004714012145996,
'CatBoost_BAG_L2/T1': 0.006834983825683594,
'CatBoost_BAG_L2/T2': 0.12855005264282227,
'ExtraTrees_BAG_L2/T1': 0.10438394546508789,
'WeightedEnsemble_L3': 0.0002639293670654297},
'num_bag_folds': 8,
'max_stack_level': 3,
'model_hyperparams': {'RandomForest_BAG_L1/T1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'CatBoost_BAG_L1/T1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'ExtraTrees_BAG_L1/T1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'WeightedEnsemble_L2': {'use_orig_features': False,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'RandomForest_BAG_L2/T1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'CatBoost_BAG_L2/T1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'CatBoost_BAG_L2/T2': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True},
'ExtraTrees_BAG_L2/T1': {'use_orig_features': True,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True,
'use_child_oof': True},
'WeightedEnsemble_L3': {'use_orig_features': False,
'max_base_models': 25,
'max_base_models_per_type': 5,
'save_bag_folds': True}},
'leaderboard': model score_val pred_time_val fit_time \
0 CatBoost_BAG_L2/T1 -42.143245 0.140617 3.158673
1 WeightedEnsemble_L2 -43.369714 0.134118 0.521992
2 WeightedEnsemble_L3 -43.407207 0.437027 1016.738522
3 CatBoost_BAG_L2/T2 -43.751974 0.262332 1015.665507
4 RandomForest_BAG_L1/T1 -44.340259 0.065923 0.276719
5 ExtraTrees_BAG_L2/T1 -44.938209 0.238166 0.802502
6 RandomForest_BAG_L2/T1 -45.322988 0.203829 1.087548
7 CatBoost_BAG_L1/T1 -45.454415 0.105544 505.513521
8 ExtraTrees_BAG_L1/T1 -45.547377 0.067859 0.169706
pred_time_val_marginal fit_time_marginal stack_level can_infer \
0 0.006835 2.712248 2 True
1 0.000336 0.075567 2 True
2 0.000264 0.075816 3 True
3 0.128550 1015.219082 2 True
4 0.065923 0.276719 1 True
5 0.104384 0.356077 2 True
6 0.070047 0.641123 2 True
7 0.105544 505.513521 1 True
8 0.067859 0.169706 1 True
fit_order
0 6
1 4
2 9
3 7
4 1
5 8
6 5
7 2
8 3 }
# Output the fit summary of the training run
predictor_new_hpo.leaderboard(silent=True).plot(kind="bar", x="model", y="score_val")
<AxesSubplot:xlabel='model'>
# Remember to set all negative values to zero
predictions = predictor_new_hpo.predict(test)
predictions = predictions.clip(lower=0)
# Same submitting predictions
submission_new_hpo = pd.read_csv('data/sampleSubmission.csv', parse_dates=['datetime'])
submission_new_hpo["count"] = predictions
submission_new_hpo['count'] = submission_new_hpo['count'].apply(lambda x: round(x))
submission_new_hpo.to_csv("submission_new_hpo.csv", index=False)
submission_new_hpo.head()
| datetime | count | |
|---|---|---|
| 0 | 2011-01-20 00:00:00 | 15 |
| 1 | 2011-01-20 01:00:00 | 7 |
| 2 | 2011-01-20 02:00:00 | 3 |
| 3 | 2011-01-20 03:00:00 | 2 |
| 4 | 2011-01-20 04:00:00 | 2 |
!kaggle competitions submit -c bike-sharing-demand -f submission_new_hpo.csv -m "new features with hyperparameters"
100%|████████████████████████████████████████| 149k/149k [00:02<00:00, 63.7kB/s] Successfully submitted to Bike Sharing Demand
!kaggle competitions submissions -c bike-sharing-demand | tail -n +1 | head -n 6
fileName date description status publicScore privateScore --------------------------- ------------------- --------------------------------- -------- ----------- ------------ submission_new_hpo.csv 2022-04-04 20:10:27 new features with hyperparameters complete 0.51299 0.51299 submission_new_hpo.csv 2022-04-04 18:08:27 new features with hyperparameters complete 0.50650 0.50650 submission_new_features.csv 2022-04-04 17:33:48 new features complete 0.50239 0.50239 submission.csv 2022-04-03 22:23:11 first raw submission complete 1.80656 1.80656
0.50650¶# Taking the top model score from each training run and creating a line plot to show improvement
# You can create these in the notebook and save them to PNG or use some other tool (e.g. google sheets, excel)
fig = pd.DataFrame(
{
"model": ["initial", "add_features", "hpo"],
"score": [-51.0252, -41.6628, -42.143245]
}
).plot(x="model", y="score", figsize=(8, 6)).get_figure()
fig.savefig('model_train_score.png')
# Take the 3 kaggle scores and creating a line plot to show improvement
fig = pd.DataFrame(
{
"test_eval": ["initial", "add_features", "hpo"],
"score": [1.80656, 0.50239, 0.50650]
}
).plot(x="test_eval", y="score", figsize=(8, 6)).get_figure()
fig.savefig('model_test_score.png')
# The 3 hyperparameters we tuned with the kaggle score as the result
pd.DataFrame({
"model": ["initial", "add_features", "hpo"],
"hpo1": ["default", "default", "CAT: {'iterations': 10000}"],
"hpo2": ["default", "default", "RF: {'n_estimators': 300}"],
"hpo3": ["default", "default", "XT: {'n_estimators': 300}"],
"score": [1.80656, 0.50239, 0.50650]
})
| model | hpo1 | hpo2 | hpo3 | score | |
|---|---|---|---|---|---|
| 0 | initial | default | default | default | 1.80656 |
| 1 | add_features | default | default | default | 0.50239 |
| 2 | hpo | CAT: {'iterations': 10000} | RF: {'n_estimators': 300} | XT: {'n_estimators': 300} | 0.50650 |